2 research outputs found
Physics-informed reinforcement learning via probabilistic co-adjustment functions
Reinforcement learning of real-world tasks is very data inefficient, and
extensive simulation-based modelling has become the dominant approach for
training systems. However, in human-robot interaction and many other real-world
settings, there is no appropriate one-model-for-all due to differences in
individual instances of the system (e.g. different people) or necessary
oversimplifications in the simulation models. This requires two approaches: 1.
either learning the individual system's dynamics approximately from data which
requires data-intensive training or 2. using a complete digital twin of the
instances, which may not be realisable in many cases. We introduce two
approaches: co-kriging adjustments (CKA) and ridge regression adjustment (RRA)
as novel ways to combine the advantages of both approaches. Our adjustment
methods are based on an auto-regressive AR1 co-kriging model that we integrate
with GP priors. This yield a data- and simulation-efficient way of using
simplistic simulation models (e.g., simple two-link model) and rapidly adapting
them to individual instances (e.g., biomechanics of individual people). Using
CKA and RRA, we obtain more accurate uncertainty quantification of the entire
system's dynamics than pure GP-based and AR1 methods. We demonstrate the
efficiency of co-kriging adjustment with an interpretable reinforcement
learning control example, learning to control a biomechanical human arm using
only a two-link arm simulation model (offline part) and CKA derived from a
small amount of interaction data (on-the-fly online). Our method unlocks an
efficient and uncertainty-aware way to implement reinforcement learning methods
in real world complex systems for which only imperfect simulation models exist
Neuromuscular Reinforcement Learning to Actuate Human Limbs through FES
Functional Electrical Stimulation (FES) is a technique to evoke muscle
contraction through low-energy electrical signals. FES can animate paralysed
limbs. Yet, an open challenge remains on how to apply FES to achieve desired
movements. This challenge is accentuated by the complexities of human bodies
and the non-stationarities of the muscles' responses. The former causes
difficulties in performing inverse dynamics, and the latter causes control
performance to degrade over extended periods of use. Here, we engage the
challenge via a data-driven approach. Specifically, we learn to control FES
through Reinforcement Learning (RL) which can automatically customise the
stimulation for the patients. However, RL typically has Markovian assumptions
while FES control systems are non-Markovian because of the non-stationarities.
To deal with this problem, we use a recurrent neural network to create
Markovian state representations. We cast FES controls into RL problems and
train RL agents to control FES in different settings in both simulations and
the real world. The results show that our RL controllers can maintain control
performances over long periods and have better stimulation characteristics than
PID controllers.Comment: Accepted manuscript IFESS 2022 (RehabWeek 2022